Maintaining a relationship between two different items of data

ABSTRACT

Data is stored persistently. At least two different items of the data are stored in two different non-conflicting regions or two different physical clusters. A relationship is maintained between the two different items of data. The relationship enables a process to reach any one of the data items from the other data item. Consistency of the relationship is maintained notwithstanding updates of either or both of the items.

RELATED APPLICATIONS

This application is a continuation (and claims the benefit of priorityunder 35 USC 120) of U.S. application Ser. No. 12/535,834, filed Aug. 5,2009, which is a continuation of U.S. application Ser. No. 10/385,758,filed Mar. 11, 2003 (now U.S. Pat. No. 7,587,428), which is acontinuation of U.S. application Ser. No. 09/687,268, filed Oct. 13,2000 (now abandoned). This application is also a continuation of U.S.application Ser. No. 14/561,281, filed Dec. 5, 2014, which is acontinuation of U.S. application Ser. No. 13/828,209, filed Mar. 14,2013 (now U.S. Pat. No. 8,935,225), which is a continuation of U.S.application Ser. No. 12/711,402, filed Feb. 24, 2010 (now U.S. Pat. No.8,489,567), which is a continuation of U.S. application Ser. No.10/821,586, filed Apr. 9, 2004 (now U.S. Pat. No. 7,689,560), which is acontinuation of U.S. application Ser. No. 09/687,941, filed Oct. 13,2000 (now abandoned). The contents of the above-referenced applicationsand patents are incorporated herein in their entirety by reference.

FIELD

This invention relates to persistent data storage techniques.

BACKGROUND

A large-scale database system may contain millions of records that areaccessible to millions of users. Potentially, tens of thousands of dataaccesses on the records may take place every second. The database systemmay include data storage devices accessed by processes running onmultiple processors. The storage devices and processors can bedistributed in various locations connected via networks. For example, alarge retail business could have a first storage device that maintainsnames and addresses of its customers, a second storage device thatmaintains inventory lists, and a third storage device that maintainspurchasing history of its customers. The first storage device is locatedin Boston, the second one in Los Angeles, and the third one in Chicago.Each storage device is managed by a different processor, which isconnected to the others by a wide area network (WAN). When a customerLisa places an order for a coffee table, for example, through a clerk ina call processing center operated by the retail business, the clerk hasto check, via the WAN, if the coffee table is available from the storagedevice in Los Angeles. The clerk may also need to access the storagedevices in the other locations to retrieve Lisa's address for shippingand update her purchasing history. At the same time, another customerRobyn may place an order for the same coffee table through another clerkin the call processing center. Both clerks will be reading from the samestorage device and trying to update the same inventory record for thecoffee table.

In the above example, the three different storage devices containdifferent types of data records that usually can be accessedindependently. Using multiple processors, as in the above example, canimprove the performance of the database system in terms of throughputand load-balancing, as long as data accesses are independent and eachaccess can run on a different processor in parallel.

Because a distributed database system is accessible by multipleprocesses, conflicts may occur if the processes are not properlycoordinated. Examples of conflicts include: two processes attempting toupdate the same record at the same time with two different values (as inthe coffee table example); a process attempting to read a record that isbeing deleted by another process; and a process attempting to update arecord that links to a related record being updated by another process.When a conflict happens, the operations of processes that access thesame or related data records may interleave in an unpredictable way,such that the results of the operations may be incorrect and may destroythe data consistency of the database system.

One approach for resolving conflicts uses a semaphore that locks a datapiece (e.g., a variable, a customer record, or a department database)when a process is accessing a data entry within the data piece, andreleases the lock when the process finishes the access. All otherprocesses must check this semaphore before accessing the data piece tosee if any process is currently using it. This approach may requiremillions of locks on millions of data pieces if the granularity of datapieces that can be locked is small, or may block large numbers ofaccesses if the granularity of data pieces is large, because locking anentire department database, for example, prevents efficient parallelexecution of jobs that access disjoint data sets that happen to bestored in the same department database.

In addition to conflicts, a large-scale database system may also sufferfrom inefficient data access. To avoid searching the entire databasesystem just to locate a data record in a storage device, a summaryinformation (e.g., a table of content, an index, or a cross-reference)of data records is usually provided in an easy-to search format.However, the summary information may be subject to corruption unless itsconsistency with the data records is always enforced. Furthermore, thetasks of updating the summary information may also create conflicts, andtherefore must be scheduled effectively.

SUMMARY

In general, the invention features a method that includes storing datapersistently, storing at least two different items of the data in twodifferent non-conflicting regions or two different physical clusters,maintaining a relationship between the two different items of data, therelationship enabling a process to reach any one of the data items fromthe other data item, and maintaining the consistency of the relationshipnotwithstanding updates of either or both of the items.

Other features and advantages of the invention will become apparent fromthe description and the claims.

DESCRIPTION

FIG. 1 is a diagram illustrating a data processing center using anupdate stream processor;

FIG. 2 is a diagram of a federated database;

FIG. 3 is a diagram of an update stream processor;

FIG. 4 is a diagram illustrating an alternative design for an updatestream processor;

FIG. 5 illustrates an index entry;

FIG. 6 illustrates a user interface for a class editor;

FIG. 7 illustrates a display of a schema;

FIG. 8 is an example showing the process of modifying an index whenloading a file;

FIG. 9(A)-(D) illustrate the process for establishing a relationship;

FIG. 10 is flowchart of the process for establishing a relationship;

FIG. 11 illustrates a user interface for selecting cache variables for arole;

FIG. 12 is flowchart of the process for deleting a relationship;

FIG. 13( a)-(f) illustrate the sequence of messages sent among threeroles for deleting a relationship.

Referring to FIG. 1, a data processing center 191 includes a transactionsystem 192, a Business Data Unit (BDU) 22, and an update streamprocessor (USP) 23. Transaction system 192 is accessible via networksincluding a public network 195 (such as the Internet) and a local areanetwork (LAN) 181 by potentially millions of users, who may be forexample, customers with computers 189 or call center operators 199 of alarge retail business that operates data processing center 191. Theusers submit requests, which may be merchandise orders or addressupdates, for example, through their respective workstations.

Transaction system 192 includes one or more servers 196 that run anapplication program (not shown) that communicates with the workstations,receives requests from the users, and automatically translates therequests to tasks or job instructions 198. A request, for example, maybe a purchase order of a blue sweater for a person named Bill. A requestis in a pre-defined electronic format, and a job instruction 198 is in aform recognizable to processes in USP 23 that create jobs. The processesthat create jobs for USP 23 are called job creation processes (JCPs)350, or producers.

A job created by JCP 350 is in the form of a job object. A job objectincludes a data structure that points to one or more objects stored inBDU 22. The job object also contains instructions executed by the jobexecution process (JEP) that act on the BDU objects. Because there is aone-to-one relationship between a job and a job object, we will refer,hereafter, to a job object as a job.

A task is like a job in that it is also an object that containsinstructions to be executed by the JEP but it does not necessarily pointto objects stored in the BDU. A task can spawn jobs if necessary and cansend an acknowledgment back to the application program when the task andall spawned jobs are complete. If the task is to provide anacknowledgement then the mechanism and necessary parameters fortransmitting the acknowledgment are recorded in the task object. A taskcan also provide an acknowledgement that it has been received and isguaranteed to execute.

As an important step in making sure that the execution of one job willnot conflict with the execution of another job, the application programof transaction system 192 assigns the job an integer called a contentionindex, included in job instruction 198. Each contention index representsa pre-partitioned disjoint data set of BDU 22, e.g., a data set 180. Thepre-partitioning uses an algorithm defined before any objects are addedto BDU 22. The algorithm is designed to achieve optimal load-balancingfor job executions on the BDU objects. Tasks can be assigned toarbitrary contention spaces since they do not access the BDU objectdirectly.

Within each data set 180, BDU objects relate to one another in the sensethat when a JEP 300 accesses an object in a data set, conflict may occurif another process accesses another object in that data set. Jobs of thesame contention index may require related objects in the same data set180 to be accessed and therefore must be executed in serial; jobs ofdifferent contention indexes can be executed in parallel (concurrently)to increase throughput.

A large job may be divided into one or more steps. For example, supposea job loads a bulk file containing one million records in BDU 22. Thejob may be divided into one million steps, with each of the stepsloading one of the one million records. Typically, there are not a lotof computations in a step; therefore a step can be executed in a smallfraction of time compared to execution time for the entire job. The jobis responsible to maintain enough state, which includes updating avariable containing the file position after every step, to ensurecontinuous operations after a fault. Periodically, but between steps,JEP 300 commits a transaction containing the results of the completedsteps, and begins a new transaction. A transaction is committed when theresults of the completed steps are successfully written and stored intoBDU 22. During the time JEP 300 commits the current transaction, thestate of the running job including the file position is updated. If afault occurs, the job would have enough information to position the fileto the last recorded position in a recovery procedure.

An existing job may require new jobs to be spawned by JEP 300. Thespawned jobs in the sweater example may include updating the monthlygross revenue for the clothing department and updating the inventory forthe blue sweater. After a job is spawned by JEP 300, the job is loadedinto USP 23. To maintain consistency of the database, all jobs spawneddue to the execution of a job J will be added in the same transaction toa staging cell in the contention space in which job J intends itsspawned jobs to produce effects. The staging cell and the contentionspace will be described later.

USP 23 manages the flow of jobs, directing them to JEPs 300 forexecution at a suitable time. The flow is managed to achieve highoverall system throughput and data processing efficiency, and to assurethat jobs executed at the same time do not conflict. With multipleprocessors running concurrent processes, USP 23 is responsible forkeeping as many processes busy as possible, while avoiding simultaneousaccess to objects in a given data set 180 by multiple processes. Toenable parallel execution without conflicts, jobs accessing the samedata set 180 are placed into a specific queue 184 assigned to one of theJEPs 300. Because there are usually more data sets than queues, a givenqueue 184 may contain jobs that access more than one data set. The queueto which a job is assigned is calculated from the job's contentionindex. For example, suppose each queue is assigned an integer numberfrom 0 to N-1, where N is the number of queues. A job with contentionindex Q will be assigned to the queue having the assigned number (Qmodulo N). Thus, the potentially large number of contention indexes canbe mapped to the relatively smaller number of queues.

Each queue can be viewed logically as a column. Within the column therecould be jobs accessing the same data set 180, that is, jobs that mayconflict. Assigning the potentially conflicting jobs the same contentionindex maps them to a given queue 184 executed by a single JEP 300. Thus,the jobs are guaranteed to be executed in serial, and therefore noconflict can happen.

On the other hand, in order to increase efficiency for processes thatproduce jobs, USP 23 is also logically organized into rows 304, each ofwhich is illustrated in FIG. 1 as a stripe over all the queues. Each rowhas a row control object that can be locked to indicate that the row isbeing accessed by a process. A write lock is requested when a processwishes to add jobs to a row. The row can alternatively be read locked byJEP 300 when it wishes to fetch jobs in the row for execution. Theoperations of adding and fetching jobs using the locks will be describedlater. If enough rows are provided, it will at all times be possible tofind an unlocked row without waiting for one to become available.

After jobs are produced, they are loaded into one row at a time. Aproducer of jobs must find an unlocked row, lock the row, load the jobsinto the row, and then release the lock. Within row 304, jobs are placedinto queues determined by their respective contention indexes. In thismanner, all producers may write jobs into the queues at the same timewithout causing conflicts as long as there are enough rows.

In some implementations, USP 23 and BDU 22 are part of a databaseorganization called a federated database (Objectivity/DB Administration,Release 5, February 1998, Objectivity Incorporated). Referring to FIG.2, a federated database 10 contains a number of database units (twounits 100 and 110 are shown). Each database unit has a number ofcontainers 120, 130, and 140. Federated database 10, database units (100and 110), and containers (120, 130 and 140) are basic constructs of adistributed, scalable object database called Objectivity/DB®, which iscommercially available from Objectivity Incorporated.

Federated database 10 is the highest level in the Objectivity/DB®logical storage hierarchy. Although federated database 10 appears inFIG. 2 as one entity, it can be distributed across multiple data storagedevices in different locations that are connected via networks.

Physically, federated database 10 exists as a federated database file(not shown). Federated database 10 contains a system database 12, whichstores a schema 15 for federated database 10, as well as a catalog 13 ofthe additional databases 100, 110 that make up the federation. Federateddatabase 10 is assigned a unique integer that identifies it toObjectivity/DB® processes (not shown), e.g., a lock server process (aservice that Objectivity/DB® clients connect to for coordinating lockingof objects in databases).

Each database 100, 110 is at the second highest level in theObjectivity/DB® logical storage hierarchy. Database 100 stores a userapplication's persistent data, for example, customer address data for aretail business. Database 100 is physically represented by a databasefile (not shown). Each database is attached to exactly one federateddatabase and is listed in that federated database's catalog 13. Databasefiles and their associated federated database file may reside ondifferent machines. In addition to having a physical filename, database100 also has a system name, which can be specified by a system managerof federated database 10. The system name of database 100 is a logicalname within federated database 10.

The containers 120 within a database 100 hold fundamental units ofpersistent data called objects (145, for example). A container 120determines the physical clustering of objects. A container 120 is alsothe fundamental unit of locking—when any object in a container 120 islocked, the lock is applied to the entire container, effectively lockingall objects in the container.

The container-level granularity can benefit overall performance, becausea lock managing process only needs to manage relatively fewcontainer-level locks rather than potentially millions or billions ofobject-level locks. FIG. 2 shows that objects may be clustered inseparate containers and yet still reference one another (148).

For example, data set 180 of FIG. 1 and FIG. 2 may contain a number ofBDU databases 100, and each of the BDU databases 100 may contain tens ofthousands of BDU containers 120. Each BDU container 120 stores objects145 holding personal or business records, as well as links 148 betweenthe objects.

Alternatively, object 145 may represent a job performing a writeoperation, such as creating, deleting, or modifying an object in BDU 22.The BDU object receiving (i.e., affected by) the write operation musthave the same contention index as object 145. On the other hand, a jobperforming a read operation as part of its activity can read from anydatabase. A mechanism for managing read operations without conflictingwith a write operation is readily available from Objectivity MROW¹(multiple readers one writer).

FIG. 3 shows an embodiment of a federated database 10 that includes asystem database 12, a BDU 22, and a USP 23. USP 23 is organized as amatrix having (n+1) logical columns and (m+1) logical rows. The numberof columns and rows required for USP 23 to avoid conflict at all timeswill be described later.

A logical column of USP 23 and corresponding data sets 180 of BDU 22form a database (201, 202, . . . 20 n), with each database representinga contention space (211, 212, . . . 21 n). One of the logical columns,the leftmost column in FIG. 3, is stored in a root database 24. Eachlogical column, except for the one representing root database 24,includes a logical cell called an execution cell (EC), and m logicalcells called staging cells (SC).

A logical row 304 of USP 23 is a logical unit for managing the accessesto the row's constituent cells. In FIG. 3, row 304 holds staging cellsSC₁₂, SC₂₂, . . . SC_(n2).

Each logical cell, whether an execution cell or a staging cell, is acontainer that holds job objects. A staging cell is where JCP 350 placesa job after the job is created, and is also where JEP 300 receives jobsfor transferring to an execution cell. The execution cell holds readyjobs, running jobs, and waiting jobs. The staging cell holds jobs thatare loaded from JCP 350.

Root database 24 includes a Job Scheduler (JS) container and m rowcontainers (R₁, R₂, . . . R_(m)). Each row container has a row controlobject 292 that keeps a list of the constituent staging cells of therow. Row control object 292 is used as a handle for a write lock or anumber of read locks for the row. A list of constituent cells for eachcontention space is kept in a contention space object 291 stored in theexecution cell container of that contention space. The information aboutall of the row control objects 292 and contention space objects 291 iskept in the JS container.

Databases (201, for example) are located in data storage devices (e.g.,disks 311) accessible by respective processors (e.g., processor 321).Preferably, each column database is stored on a separate disk, and eachprocessor runs only a single JEP 300. For example, database 201 resideson a disk 311 accessible by a processor 321 running a JEP 300. Thisarrangement keeps network traffic low and reduces disk thrashing, thusimproving the network latency and increasing the throughput.

The physical placement of root database 24 is not critical to theperformance, because the containers in root database 24 are infrequentlyread or written.

JEPs 300 may be assigned to process jobs in logical columns of USP 23using a one-to-one mapping, i.e., one JEP per column. However, othertypes of mapping can be implemented to allow scalability andload-balancing. For example, allowing one JEP for multiple columns mayenhance the scalability of USP 23 in terms of the number of processors,processes or columns. The arrangement of one JEP for multiple columnshas a benefit that when the number of processors changes, the number ofcolumns in the USP and the number of JEPs per processor can stay thesame, and therefore requires less effort to scale the number ofprocessors used by USP 23. Furthermore, job loads may be balanced outacross multiple columns assigned to the same JEP, multiple JEPs runningon the same processor, or a combination of both. On the other hand,allowing multiple JEPs per column may improve the performance of USP 23.With the arrangement of multiple JEPs per column, only one JEP isdesignated as the execution process to prevent conflict while other JEPsonly provide assistance (e.g., pre-fetching jobs) to speedup theexecution.

To avoid all contention and assure that no process waits on a lockedrow, at least C+P rows and C columns are required for a USP having CJEPs and P JCPs. C columns are required to assure that each JEP has jobsavailable at a given time. C+P rows are required so that every JCP andevery JEP can find a row available at any given time to load new jobsinto. Taking into account the JS container, the row containers, and thecolumn containers, the total number of containers required to avoidcontention and eliminate waiting on locks is (C+P+1)(C+1). Because noprocess needs to wait on locks to load new jobs, the new jobs areaccepted by USP 23 as soon as they are produced or spawned.

USP 23 can be implemented in a number of computer languages, includingVisualWorks Smalltalk, Java or C++. Example implementations require amoderate speed network connecting several machines, with each of themachines having physical disks and processors. Each machine's disks holdcolumns of USP 23 that are accessible to that machine's processors.

In operation of USP 23, JEP 300 represents a consumer process thatexecutes and then deletes jobs in USP 23. Either periodically or whenthe JEP's execution cell has no job ready to be executed, JEP 300 scansthe rows using a round robin scheme from a random permutation of therows. If the selected row cannot be locked, the next row selected by thepermutation is attempted until a read lock is acquired on one of therows. After a read lock is acquired, JEP 300 fetches all jobs in thestaging cell located at the locked row within the designated contentionspace, copies the jobs to the execution cell, and deletes the jobs fromthe staging cell. JEP 300 then releases the read lock and beginsexecuting one job at a time. After executing a job, but in the sametransaction, JEP 300 deletes the job from the execution cell.

During the job execution, JEP 300 uses the information carried with thejob to determine if the job requires any new jobs to be spawned. The newjobs spawned by JEP 300, if any, are stored in the staging cells of arow acquired by the JEP with a write lock. The staging cells are locatedin the contention space specified by the contention indexes of the newjobs.

A row control object (292, for example) may have multiple read locksacquired by multiple consumers at the same time, as long as each readlock is acquired by a different consumer in a separate contention space.However, row control object 292 allows only one write lock at a time,which is achieved through the default Objectivity/DB®. A write lock on arow also excludes any attempt to obtain a read lock on the same row,because read and write at the same time may create data inconsistency.Similarly, the existence of one or more read locks on a row prevents theacquisition of a write lock on the same row.

JEP 300 writes back results of job execution to a persistent memory,such as a cache memory or a disk, when committing a transaction. Thetransaction of a job execution is defined based on a pre-determinedcriterion, such as duration of execution time or number of executedsteps. When the pre-determined criterion is met, for example, 10 secondshave passed since the beginning of the transaction or 500 steps of oneor more jobs have been executed, JEP 300 commits a transaction. Atransaction may include the execution of multiple jobs if the jobs areshort. For example, one transaction might include the last half of ajob, ten complete jobs, and the first half of another job.

The operations of a consumer process generally include:

1. Begin by JEP 300 selecting a job from the execution cell and sendingit a #start: message. The job responds by returning a first memento,which is an object, to JEP 300. The first memento will be passed back tothe job later. The first memento is transient (i.e., kept in RAM onlyand not stored anywhere in the federated database), and JEP 300 keepstrack of it automatically.

2. Periodically, JEP 300 asks the job if it is over by sending it an#atEnd: message and passing the current memento back to the job. If thejob returns a “true” indicator, a finish message is sent as explainedbelow.

3. If the job does not return a “true” indicator, JEP 300 sends the joba #step:withScheduler: message, passing the job the current memento andinformation stored in the JS container. The job returns a second memento(which may be the same object as the first memento). Administrativeinformation, such as the information stored in the JS container, is alsopassed to the job. The information is used if the job requires more jobsto be spawned.

4. JEP 300 then decides whether to commit a transaction of the jobaccording to, for example, whether 10 seconds have passed since the lasttransaction was committed. Then JEP 300 again asks the job if it isover.

5. Once the job returns a “true” indicator, JEP 300 sends the job a#finish: message, passing the job the current memento. JEP 300 thendeletes the job.

6. JEP 300 proceeds with the next job in the execution cell. If no jobis ready to run in the execution cell, JEP 300 scans rows in its columnfor new jobs.

Execution of a job may be interrupted by a JEP failure, causing the jobto be only partially executed. However, the state of the job can berecovered at least up to the time of the most recent committedtransaction, because the contention space object 291 records the stateof current running jobs in its execution cell container every time atransaction is committed.

The recovery procedure includes starting up a new JEP to replace thefailed one, and then informing the partially executed job to restart.The recovery procedure allows the job to reset its external state ifthere is any. The recovery procedure is generally as follows:

1. Send the job the #restart: message. The job returns a memento for thenew JEP to use in continuing execution of the job.

2. Continue at step 2 of the job execution procedure as described in theprevious section.

To add a job to USP 23, a job-producing process scans the rows using around robin scheme from a random permutation of the rows, until a writelock is successfully acquired on one of the rows. The job producingprocess can be JCP 350, or JEP 300 that is spawning new jobs. Thejob-producing process holds the write lock while the job and other jobsthat are being loaded at the same time are placed into the staging cellswithin that row, until a job-producing transaction is over. Thejob-producing transaction can be defined similarly to a transaction of ajob consumer. After the transaction is completed, the job-producingprocess releases the write lock and the jobs may be selected forexecution by the respective JEPs 300 using read locks on the rows. Thus,the operations of adding a job into USP 23 generally include:

-   -   1. Acquiring a write lock on a row by acquiring a write lock on        the row control object 292 of the row.    -   2. Adding jobs to the appropriate cells of the locked row,        according to the jobs' contention indexes.    -   3. Releasing the write lock.

The following procedure can be used to create a USP with a name“SampleUSP”.

UpdateStreamProcessor createWithName: ‘SampleUSP’ producers: 4consumers: 10.

The procedure creates 11 databases, named “UpdateStreamProcessorSampleUSP root”, “UpdateStreamProcessor SampleUSP contention space 1”,“UpdateStreamProcessor SampleUSP contention space 10”. The root databasehas a JS container and one row container for each of 10+4=14 rows. Eachof the other ten databases represents a contention space, preferablystored on a disk at or near the processor assigned to process thatcontention space.

The following example shows an instruction of an application program,for locating the USP named SampleUSP and receiving a handle to the USP.The application program, for example, may be the one stored intransaction system 192 in FIG. 1.

js:=UpdateStreamProcessor named: ‘SampleUSP’.

The above function must be called within a transaction. Once the handleis received, the application program may further instruct the processesof the USP to schedule new jobs and execute existing jobs.

The following instruction triggers a JCP 350 to lock a row and write ajob into the row.

js currentOutputRow addJob: aJob.

The currentOutputRow is a function that finds an unlocked row, and thefunction is called within a job-producing transaction. Only the firstrequest for currentOutputRow in a new transaction causes JCP 350 to findanother unlocked row; repeated requests cause JCP 350 to return the samerow.

Sometimes, jobs must be executed in a pre-determined order to ensurecorrectness of the results. A method of enforcing the pre-determinedorder of job execution is called synchronization. In a commercialdatabase system, for example, there may be relationships among persons,and these persons and the associated objects may refer to each other viaattributes. A proper order of job execution is required when updatingthe attributes, relationships, and links that relate one record toanother record or person. Otherwise, the integrity of the databasesystem may be destroyed and data consistency may be lost.

A job has a quorum fraction and a tag, both of which are used forsynchronization. A job participating in synchronization can be executedonly when all the other jobs participating in the same synchronizationarrive in the execution cell. Jobs that participate in the samesynchronization form a synchronous group identified by the tags of thejobs. If a job's tag is nil, it means that the job does not participatein any synchronization. If a job's tag is not nil, it is groupedtogether with other jobs with the same tag.

The quorum fraction of a job represents its proportion of a quorum insynchronization. For example, if 5 jobs need to be synchronized, each ofthe jobs is assigned a quorum fraction value ⅕. When the total fractionsof the jobs with the same tag in the execution cell reaches one, thosejobs are moved en masse from a Smalltalk dictionary in the transientmemory, to a ready-to-run list stored in the execution cell. Thedictionary holds a list of the jobs waiting in the execution cell. Thewaiting jobs are indexed by their respective tags so that jobs of asynchronous group can be easily identified. Waiting jobs are not yetready when some jobs in their respective synchronous groups have notarrived in the execution cell.

A job with a quorum fraction of zero is invalid. An error occurs if thetotal quorum fraction of a group of jobs that need to be synchronized isgreater than one.

Jobs of a synchronous group must be executed in the same contentionspace. If jobs in different contention spaces need to be executed in acertain order, token jobs can be generated to pad the quorum fraction ina given contention space to one. For example, suppose job 1 creates jobs2 and 3, which are all in different contention spaces. Let us furthersuppose that job 3 must executed only after job 2 has completed. Whenjob 3 is created, it is given a quorum fraction of ½, and a generatedunique tag. When job 2 is created, it has no tag, but it knows what job3's tag is. When job 2 executes, the last thing it does is create atoken job 3 a that has the same tag as job 3, and a quorum fraction of½. Only when jobs 3 and 3 a have both arrived can they execute. Notethat job 3 a might do nothing, other than act as the trigger thatachieves a quorum fraction of 1, allowing job 3 to run.

As another example, consider a very long running job, with many stepsthat produce other jobs. Say that we don't want any of these spawnedjobs to run until the main job has completed. We must usesynchronization, because the transaction may be committed many timesbetween steps of the main job, which allows the spawned jobs to betransmitted to their target contention space. We can give each spawnedjob the minimum possible quorum fraction (2⁻³²), and record how manyjobs went to each contention space. In the last step of the main job wecan send a dummy trigger job to each contention space that we sent anyjobs to, using a quorum fraction that is 1 minus the total of the quorumfractions of jobs we sent to that contention space. Thus, only whenthese trigger jobs have been sent (which is only when the main jobcompletes), can the previously spawned jobs start executing.

The tag carried by a job is a unique integer that identifies the job aspart of a synchronous group. JEP 300 uses an associative structure inRAM to map a tag integer to a synchronous group. JEP 300 groups the jobswith the same tag together to determine the quorum. Because jobs onlyexist in the database for a limited amount of time until they areexecuted, a cyclic 64-bit counter usually suffices for the purposes ofgenerating an integer, unique for any existing synchronous group in USP23. To avoid contention on the counter, each contention space object 291maintains its own 64-bit counter for the jobs spawned by thecorresponding JEP 300. Each row control object 292 also holds a counterto construct tagged jobs created by JCP 350. The column number or therow number of the container holding the job can be incorporated toensure the uniqueness of the tag. One implementation for generating aninteger for a tag of a spawned job assigns a number from 0 to N-1 toeach of the columns of a USP having N columns. The tag integer of a jobcan be generated by taking the counter value multiplied by N, and thenadding the assigned column number of the container holding the job.Similar approaches can be used for generating a tag for a job created byJCP 350. A signed integer may be used to distinguish a synchronous groupgenerated from row control object 291 and contention space object 292.

When a tag needs to be generated, JCP 350 or JEP 300 sends a message#nextUniqueInteger to row control object 292 or to contention spaceobject 291, respectively. During the time the tag is being generated, awrite lock is acquired (and is typically already acquired by a previousrequest) in the same transaction on the row control object 292 orcontention space object 291 to prevent contention on this counter.

Instruction sent to a row control object 292 for generating a tag is:

tagInt:=js currentOutputRow nextUniqueInteger.

Instruction sent to a contention space object 291 for generating a tagis:

tagInt := (js contention spaces at: 5) nextUniqueInteger.

Using the quorum fractions and tags, the correct order of job executionsis assured. For example, suppose a job J1 in contention space #1 createsjobs J2 and J3. These jobs run in different contention spaces (saycontention spaces #2 and #3 respectively). When J2 is finished, itcreates job J4. Similarly J3 creates J5. J4 and J5 are assigned to thecontention space in which J1 executed. J4 and J5 have the same taginteger as each other, and each has a quorum fraction of ½. Thus, if J4arrives first in contention space #1, it cannot be executed until J5also arrives. Similarly, if J5 happens to arrive first, it must wait forJ4 to arrive before executing.

J4 and J5 must have the same tag, but that tag must be globally unique.Therefore, it is J1's responsibility to allocate a unique integer (e.g.,by asking the current output row for the next unique integer). J1 tellsJ2 and J3 what this integer is (note that J2 and J3 have no tag of theirown, as they don't need to be synchronized). When J2 creates J4, it setsJ4's tag to this integer. Similarly, J3 sets J5's tag to this sameinteger. J2 and J3 might also have to contain information about whichcontention space to send J4 and J5 to, if it's not apparent from therest of the data J2 and J3 contain.

Example code for an application program to trigger a JCP 350 to create apair of synchronous jobs is shown below. In the code, job1 and job2 areassigned the same contention index, the same tag, and different quorumfractions that total to 1. Both of the jobs must arrive in the executioncell of the assigned contention space before either one may be executed.

| row unique job1 job2 | row := js currentOutputRow. unique := rownextUniqueInteger. ← Optionally commit transaction job1 := BeepingJobnew. job1 contentionIndex: 123. job1 tagInteger: unique. job1quorumFraction: 2/5. row add: job1. job2 := BeepingJob new. job2contentionIndex: 123. job2 tagInteger: unique. job2 quorumFraction: 3/5.”...Do anything” row add: job2. Commit transaction

After a synchronous job group arrives in the designated execution cell,a job collapsing procedure may take place before JEP 300 executes thegroup of jobs. The job collapsing procedure reduces multiple jobs into asingle job, thus eliminating redundant jobs and simplifying repeatedjobs. When a synchronous job group becomes ready to execute, JEP 300sends a #collapseJobs: message to each of these jobs in turn, passingthe collection of jobs as the argument. If one of the jobs replies witha job instead of nil, this job will be used in place of the entiregroup. This job will typically contain all the information found in theoriginal group of jobs. The execution result of the new job isequivalent to the combined results of all the jobs in the synchronousgroup. For example, N “increment counter by one” operations can becollapsed into “increase counter by N”.

An example of using synchronous job group and job collapsing isdescribed as follows. USP 23 may run a load job that processes allrecords in BDU 22 to determine if there is a match between a givenrecord and a record stored in BDU 22. For example, the given record maybe a new record containing customer John's new address. The load jobspawns a number of match jobs, and each of the match jobs comparesspecific matching attributes, such as birthday, name, social securitynumber, or a combination of the attributes, between the given record andthe stored records.

The match jobs know which record they represent, as well as how manymatch jobs were created for the record. When a match job finds thecorresponding stored records that match the given record, it createsjobs, each of which holds one of these records, and sends them back tothe contention space that started the matching. Each new job has aquorum fraction that is 1/(M*R), where M is the number of match jobs,and R is the number of records that this match job found. Note that thetotal of the quorum fractions of replies from any match job equals 1/M.In the case that no matching records were found, a special dummy jobmust be sent to indicate this, with quorum fraction 1/M.

In the example of customer John, the match jobs produced reply jobs thatreference all of John's stored records. Only when all of these replyjobs have arrived back at the original contention space can they beprocessed. This is precisely when the sum of the quorum fractionsequals 1. At this time, the match reply jobs can be collapsed into asingle job that has the complete list of matching records. This data canbe analyzed and merged as needed, and update jobs can then be sent toeach record that needs to be modified to accommodate the changedaddress.

Tasks use synchronization of jobs to enable an acknowledgment to be sentafter all jobs that were spawned as a result of the task's execution arecompleted. All spawned jobs carry the task's contention space, a uniquetag and a fraction that when added to all other fractions contained inother jobs spawned by a given job will total to the spawning job'sfraction. In the case of jobs spawned by the task their fractions willtotal to one. A quick way to generate these fractions is to take 1divided by the number of jobs that are being spawned and multiplyingthis by the spawning job's fraction and using the resulting fraction ineach of the spawned jobs where a task's fraction is assumed to be 1.This scheme will cause the sum of all fractions across the final jobs(jobs that do not need to spawn any further jobs to do work other thanacknowledgment) to total to one. The final jobs spawn an acknowledgmentjob with the recorded contention space, tag and fraction as quorumfraction. When all the acknowledgment jobs arrive at the task'scontention space they are collapsed and executed causing theacknowledgment to be sent to the application program.

Other implementations are within the scope of the claims.

For example, instead of using a separate execution cell, jobs that donot require synchronous executions can be executed directly from thestaging cells. Synchronized jobs, however, would still have to be movedto the execution cell for execution, so that they could all be executedas a synchronous group and deleted together.

To facilitate job executions directly from a staging cell, each stagingcell has a counter that indicates the number of jobs in the staging cellwaiting to be executed. The counter may be a 32-bit counter that wrapsaround to 0 when the counter value reaches 2³²−1. When JCP 350 adds anew job into a staging cell, the counter in the staging cell isincremented. Both the addition of the job and update of the counter aredone in the same transaction.

Each execution cell also has similar 32-bit counters that indicate thenumber of completed jobs for respective staging cells. When JEP 300completes a job execution, the associated counter in the execution cellis incremented with a MROW write. The MROW semantics allow the counterto be access simultaneously by a single writer and multiple readers.Periodically, JCP 350 examines the counters in the execution cell with aMROW read. The values of the counters are used by JCP 350 to determinehow many jobs can be deleted in the respective staging cells.

When JEP 300 needs new jobs to execute, the JEP reads all jobs in thestaging cell whose counter value is greater than the execution cell'scounter value, taking into account that the counter can wrap to zerowhen it reaches its maximum value. A counter value is considered greaterthan another value if the (counter value—another value) modulo maximumsize<(maximum size/2). For example, in the case of comparing the valuesof two 4-bit counters, suppose the counter value is 9, and the othervalue is 7. Since 9−7=2, 2 modulo 16=2, and 2 is less than (16/8),therefore, 9 is greater than 7. This subtraction also wraps; forexample, (0−1) is equal to the maximum value of the counter. Theworkload for JEP 300 is reduced because the JEP never needs to modifythe staging cells.

In a certain embodiment, the USP does not even have the matrix structureshown in FIG. 1 and FIG. 3. Instead, the USP includes job databases andtheir respective processes, which communicate via TCP/IP sockets. Thelocking operations are no longer needed because the concept of rows doesnot exist in this embodiment. Referring to FIG. 3A, USP 27 includes JEPsand JCPs, each of which has a job list (25) located in the memory of thesame processor running the process. Job database 26 of a JCP storesbackup copies of jobs that are sent to JEPs; job list 25 of a JEP tracksjobs waiting to be executed. When a JCP creates a job, a copy of the jobis loaded into the JCP's job database 26 as a backup. The JCP transmitsthe job via a TCP/IP socket to an appropriate JEP whose contention spaceis specified by the job's contention index. After the JEP receives thejob, it temporarily adds the job to its job list 25 waiting forexecution.

A TCP/IP socket is a software entity that allows an application programto send and receive TCP/IP messages over a network. Using the TCP/IPsockets, jobs may be sent and received as TCP/IP messages, thus hidingnetwork details from the programmers of the system.

Each JCP has a socket connection to each JEP, through which it cantransmit jobs that must be executed by that JEP. Jobs from a particularJCP destined for a particular JEP are all transmitted through the samesocket connection, and are assigned consecutive job ID numbers, modulo2³².

USP 27 utilizes the concept of an “autonomous partition” implemented byObjectivity/DB®. An autonomous partition is basically a subset ofdatabases of a federated database. Each database belongs to exactly oneautonomous partition. In this variation of the USP, each process canoperate in its own autonomous partition. Database writes can beconstrained as local to the database controlled by its associatedexecuting process, thus greatly reducing network traffic and safelyisolating failure of any processor until that processor is recovered. Asa result of reduced network traffic, the autonomous partitions alsoreduce the undesired effects of deployment on a Wide Area Network (WAN)that spans across distant geographic sites. The undesired effectsinclude higher cost of transmitting data and higher expected failurerate of communication links, as compared to a Local Area Network (LAN).Because of the reduced network traffic, the autonomous partitions notonly lower the cost for deployment on the WAN, but also lessen thedemand for reliability in the transmissions.

The TCP/IP socket connections between the JCPs and JEPs are of the“stream” variety, rather than “datagram”. The underlying networkprotocol for the “stream” variety ensures delivery of the messages,including error correction and retransmission as necessary. IndividualIP packets may arrive at the physical network adaptor in any order, zeroor more times, and arbitrarily corrupted. The “stream” socketimplementation is responsible for correctly reordering these packets,requesting retransmission of garbled packets, and discarding redundantpackets. If transmission of a packet cannot be accomplished andacknowledged in a reasonable amount of time and effort (typically a fewseconds), the protocol will simply notify the clients (i.e., the JCP andJEP) that the socket has been disconnected. If a socket is disconnected,the clients will periodically attempt to reconnect the disconnectedsockets. A JEP will continue to process jobs arriving from the connectedsockets while attempting to reconnect. Thus, job processing is continuedeven during recovery from a failed node or network link.

The packet size on a typical network is several kilobytes in length. Fora fixed-sized packet, the overhead of transmitting a packet is fixed.Because the size of a job is usually shorter than the size of a packet,it would be inefficient to transmit each job as a single packet.Therefore, before transmission, jobs are written into a buffer whosesize equals the packet size. The transmitting process packs as many jobsas possible into each buffer, and transmits the entire buffer in apacket to reduce wasted network traffic.

Occasionally, nearly empty packets still need to be transmitted;otherwise if the USP becomes quiescent the final jobs might never betransmitted. Thus, we set a limit on how long data can remain in abuffer prior to it being sent in a packet. If more than, for example, 10seconds has elapsed since the first job was written into a buffer, thebuffer is flushed to the socket, forcing the packet to be physicallysent. On the other hand, if we timed it relative to the last job in thebuffer, a trickle of jobs arriving every 9 seconds might keep the bufferfrom being transmitted for several minutes, despite the fact that someof the jobs had been waiting to be transmitted for a long time. The timelimit can be reduced if the USP is used in an environment that requireslower latency.

To ensure that jobs will be executed even in the event of a failure,committed jobs are always written to the JCP's job database 26 prior totransmission to a JEP via the socket. When a job is received by the JEP,we know that the job has already been committed to a JCP's database. Inthe event of a failure, the JCP will scan its job database 26 andretransmit to each JEP those jobs that may not have been executed yet.The JEP simply ignores jobs whose ID indicates the job has already beenreceived and executed.

To keep the JCP's database of jobs from growing arbitrarily large, eachJEP has the responsibility of recording the ID numbers of the mostrecently completed jobs, one number per JCP, every time it commits atransaction. These job ID numbers are counted by a RAM counter, and areused during recovery to tell which jobs have already been executed andcan be ignored. The JEP also periodically transmits to each JCP adeletion message containing the RAM counter value for that JCP. When theJCP receives the deletion message, it is free to delete every job withan ID less than or equal to the ID in the message, using wrappingarithmetic (i.e., to delete every job whose ID is equal to the ID in themessage, within 2³¹ below the ID in the message, or is more than 2³¹above the ID in the message).

A job deletion message cannot carry an ID of a job that has not beenexecuted. If the job is non-synchronized, the job must have beenexecuted to completion and committed. If the job is a synchronized job,duplication of information in the JEP is required. Prior to transmittinga job deletion message with an ID of a synchronized job, the JEP storesa copy of the job in job database 25 and commits it. Storing a copy ofthe synchronous job is necessary for recovery in the event of a failure;otherwise there would be no persistent record of the job. Theassociative structure in RAM, which is described earlier in jobsynchronization, records the mapping from each synchronization tag tothe list of jobs in the synchronous group with that tag, including thesynchronous job whose ID was transmitted in the deletion message. Atrecovery time the associative structure is rebuilt from the jobs in jobdatabase 25.

When the group's total quorum fraction reaches 1, the group is given theopportunity to collapse into a single job. If a collapse happens, thejobs of the group are deleted from the database and the associativestructure, and the single replacement job is stored in place of thegroup in a single transaction. The single job is treated as asynchronous group with a single member whose quorum fraction is 1.

Whether a synchronous group consists of several original jobs or onesingle job created by collapsing, when the group is ready to run, thetag of the group is recorded and job execution begins. When a job in thegroup completes, the job is deleted from the JEP's job database 25, andthe next job in the group is started. If it is required to commit atransaction part way through the execution of the group (e.g., to limitthe duration of the transaction), the JEP will record the tag of thegroup, as well as the pointer to the job being executed. If a crashhappens during the commit, the remaining jobs of the group will beexecuted before any other jobs. After all jobs of the group havecompleted, the next incoming job via any socket connection is processed.

Since each JCP/JEP pair uses consecutive ID numbers for its transmittedjobs, and since deletions occur in the same order as transmission of thejobs, the JEP can safely transmit only some of the deletion messages,with each message requesting a block of jobs to be deleted. When a JCPreceives a job deletion message, it deletes all jobs that have an IDless than or equal to the transmitted ID (using the wrapping arithmeticas described above). To reduce the number of job deletion messages, theJEP transmits a deletion message to a JCP only if either the ID of thedeletion message crosses a multiple of a pre-determined number (e.g.,1000), or the deletion happened more than a pre-determined length oftime (e.g., 10 seconds) ago and no new jobs from the JCP (or from anyJCP) have arrived in that time.

Without the latter condition, at most a few thousand jobs will have tobe retransmitted for each JCP/JEP pair when recovering from a JEPfailure. With the latter condition, the JCP may periodically deletecompleted jobs in its job database 26 even when no new jobs arrive. Thelength of time in the latter condition is a trade between the recoveryoverhead, the deletion overhead, and transmission cost. A shorter timeperiod allows the JCP to delete completed jobs more often, and thereforefewer jobs will be re-transmitted in case of a JEP failure. However,reducing the time limit below 10 seconds in the latter condition isprobably not worthwhile, because it would increase the number ofdeletion transactions that the JCP would have to perform. Asignificantly smaller value would waste a small amount of CPU timedealing with deletion of jobs in the JCP's job database 26. If a largervalue is used, a disadvantageous situation may arise that when a largenumber of new jobs finally arrive, the JCP may have wasted its idle timeand will now have to spend time performing job deletions even though newjobs are ready.

As an alternative perspective, consider the life cycle of a typicalnon-synchronized job J:

Suppose at some point of time, JCP#1 creates job J. Job J is assigned torun in contention space #2, because J manipulates the data in contentionspace #2. Assume that contention space #2 is under the control of JEP#2,and job J is assigned a unique ID number 123, one greater than the IDnumber of the previous job sent from JCP#1 to JEP#2.

The next time JCP#1 commits a transaction, a copy of job J will bewritten to JCP#1's job database 26. JCP#1's current ID numbers will alsobe written in the same transaction. Immediately after the transactionhas committed, J will be converted into a sequence of bytes and writteninto a buffer with other jobs bound for JEP#2. When that buffer is full,all the jobs in the buffer will be sent in a packet to JEP#2.

JEP#2 eventually receives the packet from its JCP#1-JEP#2 socketconnection. The packet is converted from a sequence of bytes into asequence of jobs, effectively reconstituting J and the other jobs. Thejobs are moved to a queue in RAM, where they are interleaved with otherjobs arriving from other sockets. The interleaving preserves therelative order of jobs coming from JCP#1.

Suppose that JEP#2 crashes while J is in the queue. JEP#2 is rebooted,and the socket connections are re-established. When the connection fromJCP#1 is re-established, JCP#1 retransmits all jobs in its job database26, including a copy of J. Some of the jobs that came before J may havealready been executed to completion by JEP#2. These jobs are transmittedanyhow by JCP#1, but JEP#2 ignores them. JEP#2 knows to ignore a jobwhen the job's ID is less than or equal to the currently completed jobID that JEP#2 stores in its job database 25. When J is received again byJEP#2, it is placed in the queue in job ID order with respect to otherjobs originating at JCP#1.

Eventually JEP#2 removes J from its queue and executes it. JEP#2increments a RAM counter that indicates it has now executed job 123(i.e., job J) from JCP#1. Many jobs may be executed prior to and after Jin the same transaction, hence the RAM counter may be incremented manytimes during a transaction.

When the transaction is committed, the current values of RAM countersare written to job database 25, together with the changes in the BDUobjects. This operation guarantees that each job affects the BDU exactlyonce. That is, if J increments a counter in an object, the counter willonly be incremented once because of J.

After certain transactions, JEP#2's RAM counter representing the currentcompleted job number from JCP#1 reaches 1005, which is greater than therequired value of 1000 to send a deletion message. The new counter valuewill then be transmitted back to JCP#1 in a job deletion message.

When JCP#1 receives a deletion message with ID=1005, it deletes all jobsin its database with an ID less than or equal to 1005 (using wrappingarithmetic, described above). Since J′s ID is 123, which is less than orequal to 1005, it will be deleted. Since there are about a thousand ormore jobs being deleted at this time, and since many of them werewritten out in a single transaction originally, the deletion typicallyrequires very few pages of job database 26 to be written back to disk.Once this transaction commits, there will be no more trace of J in anydatabase or in any processor's memory.

The only network communication that occurred between JCP#1 and JEP#2was: transmission of jobs from JCP#1 to JEP#2, and transmission of adeletion message from JEP#2 to JCP#1. Transmission of job J occurredtwice in the example only because JEP#2 crashed after the initialtransmission. The deletion message cleaned up about 1000 jobs with onepacket.

Network traffic can be reduced by compressing information transmitted onthe network. A simple compression scheme, for example, may be one thatreduces the size of a job. Because a job is an object, and each objectis an instance of some class that defines the structure and behavior ofthat object, we may define jobs as instances of different subclasses ofa class “Job”. Jobs may be created to update instances of a class“Address” or a class “Person”. Hence, a class of jobs includes jobswhose tasks are directed toward a class of objects. When the JCP encodesan instance of a class of jobs into bytes for the first time, the nameof that class is transmitted along with an encoding of the job object.The class is then added to the list of encountered classes and given aunique number. The next time an instance of this class is transmitted,the class's unique number is transmitted instead. The compression schemethus effectively reduces the overhead of transmitting a job.

To improve the efficiency of each JEP, a technique which we callOID-sorting can be used. In this technique, at the start of atransaction in which jobs are to be executed, all available jobs arefirst sorted by the unique object identifier of the object, if any, thatwill be modified by the job. If multiple objects may be modified byrunning a job, one can be chosen arbitrarily. If a job creates anobject, the identifier of the container which will contain the newobject is used for sorting. Execution of the jobs then proceeds throughthis list in order.

Because the sorted list of jobs might not be completely executed in asingle transaction, we must record enough information in the database toreconstruct the remaining jobs during recovery, should a failure occur.This information includes the first and last job id numbers of jobs inthe list, for each source of jobs (jobs are assigned unique id numbersonly relative to the JCP/JEP pair that the job is transmitted from/to).This lets us reconstruct the exact same list of jobs at recovery time,but we must also record how many of these jobs have actually beenexecuted whenever we commit a transaction. That information allowsperfect recovery from a failure. During recovery of a failed JEP we mustwait for each JCP to retransmit at least those jobs that participated inthe sorted list of jobs that was being executed at failure time.

When the entire sorted list of jobs has completed, job deletion messagescan then be sent to each JCP that provided the jobs that were executed.Sending deletion messages before this point is still reasonable, as longas the persistent counter that says where we are in the list is relativeto the end of the list, rather than the start. Otherwise, when some ofthe early jobs in the list have been deleted, they will not be resent tothe JEP at recovery time.

There are several reasons for sorting jobs by the unique objectidentifiers of the object affected by the job: Since object identifiersencode the physical location of an object so that object identifiersthat are close together numerically represent objects that are closertogether physically therefore fewer pages from the database may need tobe examined/written per transaction. Multiple writes to the same pagewill be aggregated together into a single physical write. Fewercontainers may need to be locked per transaction—the high bits of theobject identifier specify the container, and the low bits specify theobject within that container. The pages that are written at commit timehave strong physical proximity on the disk, so seek times will bereduced.

To ensure that at recovery time the exact same list of jobs is produced,the sorting criterion must break ties consistently. Thus, afterconsidering the object identifier of the object being updated, ties mustbe broken by further sorting based on the originating JCP# and the job'sid number. This pair of values is guaranteed unique, and is sufficientto unambiguously break ties (arbitrarily).

Because each change to an object can potentially cause much work to bedone (e.g. re-indexing the object as described below), we may wish toavoid this situation when possible. Thus, when a job is asked toexecute, it may examine the list of jobs that affect the same object(these jobs come after the current job in the sorted list). The changesrepresented by these jobs can then be collapsed together into a singleupdate operation, which in our example would allow re-indexing to occuronly once for this set of changes. Jobs can carry timestamps ifappropriate, to identify the order in which to perform conflictingchanges.

Besides ordering jobs based on the location of the data being modifiedby them, one may wish to prioritize jobs based on how urgently they mustbe completed. There might not be any urgency to complete a batch job,but an object-updating job triggered directly by a user should probablyrun as soon as possible. Several basic mechanisms exist to support thisneed.

In a deadline-based soft real-time priority scheme, each job hasassociated with it a time. It is strongly desirable that the jobcomplete by this time. Unfortunately, this interferes with OID-sorting.To resolve this conflict, the following algorithm is used. At any pointin time a JEP has a heap of jobs, sorted by expiration time. The jobexecution process looks at the top element of the heap. This is the jobwith the earliest deadline, possibly in the past if we're temporarilyoverloaded. Jobs are popped from the heap until we've popped either ajob more than 5 seconds in the future, or all the jobs, whichever comesfirst. We then sort these jobs in OID order and attempt to run as manyof them as possible in a transaction. If we don't finish running themall in a single transaction (because for example more than 10 secondshave elapsed in that transaction and 10 seconds is the maximumconfigured transaction time), we commit the transaction and continueexecuting these jobs in the next transaction.

To deal with deletion of completed jobs in this scheme, we look to thesolution that was already described for synchronized jobs. Asynchronized job is considered “dealt with” when a copy has beencommitted to the database of its JEP. At this time (or some timethereafter), a message is sent back to the JCP indicating that the JCPmay delete its copy of that job. To support OID-sorted execution (i.e.,execution not in job id order), we must commit copies of all jobs, notjust synchronized ones, to the JEP's database.

Referring again to FIG. 1. BDU 22 in data processing center 191 maycontain millions of objects. To locate an object in the BDU, informationabout the object, including its location or other attributes, is storedand arranged for efficient access in a parallel (concurrent) processingenvironment.

For a data processing center 191 of an insurance company, for example,each of the BDU objects may represent a record for a person insuredunder a certain type of policy. When there is a change in the featuresof that type of policy, an insurance agent may wish to locate all thepeople insured under that type of policy and notify them of the change.To efficiently locate the people, a file that includes pre-sortedentries may be used. Each of the pre-sorted entries contains a pointerto one person's object and other information that is essential inidentifying the person. For example, the insurance agent may use a filethat has entries for all the people insured under a given type ofpolicy, pre-sorted by last name.

When objects are created, deleted, or updated, the corresponding entriesin the file must be updated. To assure that all jobs that create,delete, or update objects will consistently modify the correspondingpre-sorted entries, the jobs must agree upon a common mechanism and acommon format to make necessary changes on the file, the pre-sortedentries, and the objects. The format of the file and the pre-sortedentries are designed to facilitate searching and locating a desiredobject, and therefore, the format or layout of information in apre-sorted entry is typically the same as other entries in the file.

The common mechanism pre-defines what attributes of an object are usedfor pre-sorting the corresponding entry, what information is displayedin the entry, and how changes in an object should propagate to theentry. We call the common mechanism an Asynchronous Index Manager (AIM),the file an index, and the pre-sorted entry an index entry.

In a database system that allows tens of thousands (or more) ofsimultaneous data accesses, it is crucial to maintain the integrity ofthe index while avoiding access conflicts. The AIM defines how indicesshould be structured and maintained. The task of executing the changesin the index is carried out by jobs scheduled by the USP. For example,when an object is added or deleted, new jobs are spawned to add ordelete the corresponding index entries in the appropriate indices.Similarly, when updating an object would have an effect on the accuracyof index entries, jobs are spawned to update the appropriate indicesthat contain the affected index entries.

The index is similar in concept to a card catalog used in a library forlocating specific books. The card catalog holds index cards, each ofwhich contains information about a book. The information may include abrief summary of the book, as well as other necessary information for auser of the card catalog to locate the book in the library.

Books may be looked up by any one of multiple criteria, such as byauthor, title, or subject, and the index cards representing the booksare sorted by a search criterion for efficiency. A given catalogtypically holds information for a collection of things of the same type.For example, there may be separate catalogs of books, catalogs ofperiodicals, or catalogs of audio media (e.g., tapes or CDs). All theindex cards in a catalog have the same layout in terms of how theinformation is organized; for example, the title of a book is at the topof every index card and the author's name is below the title.

The index used for locating objects in the BDU is conceptually similarto a card catalog. An index contains a collection of index entries(index cards), each of which contains a small summary of an object(book). Objects identified within an index are of the same type, i.e.,the same class in an object-oriented terminology. Index entries withinan index have the same data structure. Index entries may be sorted orhashed by a pre-defined key, depending on the intended access patternand the size of the index.

Each index has key and non-key attributes that can be defined by asystem administrator. The key attributes are used for sorting or hashingan index entry, and the non-key attributes are displayed in the indexentry together with the key attribute. The display of the non-keyattributes allows certain pre-defined information about the object to beviewed by a user of the index without having to retrieve the object fromthe BDU. In the library example, an index card sorted by the ISBN maycontain information including the book title and the author.

FIG. 5 is a diagram of an index entry. Every person in the database hasa corresponding index entry 40 in an index called Person-SSN, whichmeans the index contains a class of person objects, represented byrespective index entries sorted by the key attribute SSN. Each indexentry of the index contains the SSN, a person's first name and lastname, and a pointer to a person object 41, which in turn points to aname object 42 containing more information about the name of the person.

Indices and index entries may be stored on disks and in memory. Storinga copy of the index in memory can reduce index access time and thereforeincrease the processing speed of locating an object. The copy of theindex in memory is implemented as a memory-resident (i.e., RAM-resident)search structure (e.g., a binary search tree or hash table). When a usersubmits a request for updating a BDU object, the resulting update jobnot only updates the BDU object, but also updates the associatedindices. The search structure must be updated in lockstep with thechanges in the BDU and indices on disk. Because each index update is aconsequence of executing a job that updates a BDU object, the job isgiven an additional responsibility of maintaining the consistency ofsearch structures with BDU objects and the indices on disk. In case of aJEP failure, at recovery time the JEP rebuilds the search structure inmemory by scanning the BDU.

Modifications to a BDU object may not take place immediately after amodification request is sent, because changes in the BDU are notreflected until a transaction is committed. Modification to the memorysearch structure, however, could happen immediately. If a user submits aquery for information about an object that has not been committed to theBDU, the object cannot be located. An object identifier (OID) may nothave been assigned for such an uncommitted object. In this case, theuser may simply discard the result from the query. The situations thatupdates in database may lag behind updates in search structure maysometimes arise in a standard database system. If an object has not yetbeen written to a standard database system, we will not be able to findthe object. An alternative scheme to handle this situation is not tochange the search structure immediately when executing a job, but ratherto accumulate the changes and apply them immediately after a transactionis committed.

FIG. 6 is a user interface called a class editor 50 with which a systemadministrator may define an index for a class of objects. Generally, anobject can be categorized by an object type, such as person type orproduct type. An object type may include multiple classes; for example,a car insurance company may classify its policyholders as people withcomprehensive coverage and people with liability coverage. Each of theclasses has at least one corresponding index. Each index has a keyattribute and non-key attributes, which can be edited from the classeditor.

Class editor 50 allows a system administrator to choose a key 51 for anindex he creates or edits and to select the non-key attributes 52 hewishes to store in the index entry. In FIG. 7, the index being editedcontains a class of Test::Person 53. The key of the index is SSN, andeach index entry of the index contains information about the SSN, theaddress of the person, and the postal code for the person's address (notshown).

Since a person may have more than one address, more than one postal codemay be associated with that person. For efficiency in locating allpersons having the same postal code, where the postal code is a key inan index, multiple index entries are created for a person who hasmultiple addresses, one index entry per address.

To find out what indices are defined, a system administrator can open anobject schema window to edit and display a schema that contains thedefinitions of the indices. FIG. 7 shows an object schema window 60 thatdisplays the definitions of object classes (61, 62, and 63) and theirassociated indices and attributes. The schema contains layouts ofclasses for the objects in the database. Each class layout describes thephysical structure of instances of that class in terms of attributes andrelationships. Additionally, the schema describes how to distributeobjects among databases and processors without contention, how to parseinput files that are to be loaded into the database, and how toconsolidate data from multiple sources.

Every time a request for a task that involves adding, deleting, orupdating an object arrives at the USP, the request is sent to a JCP 350to create one or more jobs that act on the request. The JCP uses theinformation in the schema to find out which indices are defined for thatobject class, and what the keys are for the indices. JCP 350 thendetermines necessary changes to the indices, such as adding, deletingand updating index entries, and decides the sequence of jobs that needto be created in order to update the indices and to complete the task.Each requested action has a different requirement on the order in whichobjects and their respective index entries are modified. The requirementmust be strictly enforced to maintain the integrity of the indices.

FIG. 8 illustrates an example of an index modification process forloading a file 70. File 70 may require adding 610, deleting 630 andupdating 650 objects in BDU 22. For example, file 70 may containcustomer records of a new division that was just acquired by aninsurance company. The acquired customer records may contain duplicatedinformation or more up-to-date information about existing customers, orcontain information about new customers. To consolidate the acquiredcustomer records with the existing customer records, jobs are created toadd, delete, and update the BDU objects representing the customerrecords. As an example of the jobs that are created and the order inwhich they must be done, when deleting an object (630), links betweenthe object and its index entries must be deleted first (631). Then jobsare produced to delete all the index entries referring to the object(632, 633). After the index entries are deleted, another job is spawnedto delete the object (634, 635). The index entries must be deletedbefore the object is deleted; otherwise, another process may use one ofthe index entries to access the object while the object has beendeleted.

In some implementations such as Objectivity/DB®, the pointer to anobject is reused. The pointer to an object is called an objectidentifier (OID) and includes four 16-bit unsigned integers that specifythe object's database, container, page number, and page slot in thestorage. The index entry of the deleted object contains the OID of thedeleted object, but the OID may have been reassigned to another objectthat is added to the same database, container, and storage location asthe deleted object. Therefore, if an object is deleted before its indexentries, one of two error conditions may happen: either a process maytry to access a non-existent object, or the process may refer to thewrong object.

To avoid contention in deleting an object and its index entries, jobsthat carry out the deletions of an object are scheduled by the USP. Thejobs may be scattered over several contention spaces. Each of the jobscauses another “response” job to be spawned to indicate its completion.The response jobs are synchronized and loaded into the contention spacewhere the object resides. When all the response jobs arrive in theexecution cell (as determined by the completion of a quorum), all theresponse jobs are collapsed into a single job that deletes the object.

The ordering of steps for adding an object is the reverse of deletion.When adding (610) an object, the object must be created before any indexentries can refer to it. When an object is created (611, 612) and storedin a persistent memory, “insert” jobs are spawned (613), each creatingan index entry (614, 615) and each executed in an appropriate contentionspace. Note that these jobs are created in the same transaction as theobject creation; otherwise the object might end up stored without thecorresponding jobs, if a failure occurs. Then jobs are created toestablish links between the object and its index entries (617).

When updating an object, the update may have no effect on any of theobject's index entries. For example, a person's color preference may bestored in the person's object, but not in any of the index entries. Inthis situation, no update is needed for the index entries. In otherexamples, the update may require the index entries to be updated ordeleted, or require new index entries to be created. For example, if aperson's address is changed and address is part of the informationstored in the person's index entry, the index entry must be updated. Ifthe person bought another house in another postal area, and the index iskeyed (i.e. sorted) by postal code, a new index entry containing theaddress of the person's new house needs to be inserted.

In the process of updating an object, JCP 350 creates a job to updatethe object (650, 651) before updating any of its index entries. In theexample of updating a person's address, although the index entrycontains the old address before the index entry is updated, the OIDcontained in the index entry that points to the person's object is stillcurrent. Therefore, an updated object can still be located by using theold index entry. When updating an object, JCP 350 figures out andproduces a list of index entries that should exist after the update.This list is then compared with the current list of index entriesattached to the object to determine which re-indexing jobs need to beperformed, that is, which index entries should be updated (652), created(654), deleted (653), or remain unchanged.

If an index entry should be deleted (653), it is first disconnected fromthe object, then JCP 350 creates a job to delete the index entry. Thisjob sends back a reply job to the object indicating completion. Thisreply job is necessary for a wait-free algorithm described below. If anindex entry should be added, JCP 350 creates a job that contains enoughinformation to create the index entry in the appropriate contentionspace, and then sends back a response job to the object indicating theindex entry that was created. If an index entry should be updated, JCP350 creates a job that contains enough information to update theexisting index entry, and then sends back a response job to the objectindicating completion. If an index entry should remain unchanged, thereis nothing to be done.

To assure that re-indexing jobs work correctly when multiple overlappingchanges occur to an object (i.e., changes that happen before the indexentries have all been brought into agreement with the object), await-free algorithm is used. As will be described below, the wait-freealgorithm allows changes in an object while the object has outstandingjobs, and further avoids contention between all the re-indexing jobs.The object reserves a two-bit field for an index entry update operation:a re-indexing indicator and a pleaseReindex indicator. The re-indexingindicator indicates that there are outstanding re-indexing jobs thathave not yet sent back the response jobs. The pleaseReindex indicatorindicates that the object was changed before its re-indexing jobs werecompleted. Responses from the individual re-indexing jobs aresynchronized. The synchronization allows all the re-indexing responsesto collapse into a single job when all the responses are present in theobject's corresponding execution cell. The single job updates a list ofindex entries attached to the object. Immediately after the update, theobject's pleaseReindex indicator is examined. If the indicator is set,it indicates that the object has changed during the re-indexing that wasjust finished. Another re-indexing operation according to the new changewill start right away.

A request for deleting an object may arrive during a re-indexingoperation. Deletion requests have priority over update requests, becauseany updates on the object and its index entries vanish after the objectis deleted. An additional reserved two-bit field is used in the object:one is deleting, and the other one is pleaseDelete. Deleting bitindicates if the object is in the process of being deleted, andpleaseDelete indicates if there is a request for deleting the object.When either bit is set, the pleaseReindex indicator is ignored, andsubsequent requests to update the object are also ignored.

If a user only wishes to read certain information about a BDU object,the user may send a query. Queries, unlike most other jobs, do notcreate changes in objects, index entries, or indices. In the embodimentsof the USP using TCP/IP sockets, queries may be handled as query jobs toreduce the amount of data transmitted via a network. When a requestorsubmits a query for locating a BDU object, a JCP converts the query intoa query job, which is then sent to the JEP of the contention space inwhich the requested object resides. Each query job has an ID, which isused for the originating JCP to match a result with the correspondingquery. The query job is not given a sequencing number as other jobs thatare sent over the network. If the query job is lost in networktransmission on the way to a JEP, it is up to the requestor to re-submitthe query (possibly after a time-out). The handling of lost queries isreasonable for customers accessing a company's databases from the WorldWide Web using Web browsers (such as Microsoft's Internet Explorer).

When a query job is received by the JEP, instead of adding it to thequeue of ready jobs, the query job may be added to a different queue,the queue of query jobs. Between ordinary jobs, and even between thesteps of an ordinary job, this queue of query jobs may be examined. Ifthere is a query job waiting, the query is executed immediately, and theresult is sent back to the originating JCP, with the job's ID attached.Because query jobs only read data in the BDU, allowing the query toprecede other jobs does not introduce any ordering problems.

An object in the BDU may be located not only with an index, but alsowith links connecting the object to other related objects. Many BDUobjects are related to each other. For example, referring again to FIG.1, data processing center 191 of an insurance company may store itspolicyholders' objects and product objects in BDU 22. Suppose apolicyholder Bill has earthquake insurance, which means that an“ownership” relationship exists between an object representing Bill anda product object representing earthquake insurance. If a user of thesystem wishes to locate the product object owned by Bill, one way is toretrieve Bill's object, look for which insurance policy Bill has, andlocate an index entry of earthquake insurance in an index of insuranceproduct objects. Alternatively, information about the earthquakeinsurance may be retrieved by establishing a direct link between theobject of Bill and the product object of earthquake insurance. Using thedirect link, information related to an object of interest (e.g., aninsurance product object) may be retrieved directly without goingthrough an index.

The direct link between objects is called a relationship. A relationshipmay be, for example, an ownership or a parentage. Relationships betweenobjects can be built by a mechanism called an Asynchronous RelationshipManager (ARM). A system administrator only needs to define arelationship between specific classes of objects, and jobs will beautomatically created to build the relationship between thecorresponding instances of the classes (i.e., objects) according to theARM mechanism.

The ARM defines how relationships should be structured and maintainedfor a system that allows millions of simultaneous accesses, such as in alarge-scale distributed database system. The ARM provides an environmentand a set of common rules to guarantee the integrity of therelationships as objects are added, modified, or deleted across thedistributed databases.

For example, if the insurance company decides to stop carrying theearthquake insurance that Bill has, the ARM guarantees that therelationship between Bill and the earthquake insurance will beautomatically deleted before the product object of earthquake insuranceis removed from database. The task of executing the changes in therelationships is carried out by jobs scheduled by the USP to allow highthroughput and efficiency. For example, when an object is added ordeleted, new jobs are spawned to add or delete the associatedrelationships. Similarly, if updating an object requires itsrelationships to be updated, jobs are spawned to update the appropriaterelationships.

Jobs executed by JEP 300 may be jobs that add, delete, or update a BDUobject. Changes in the object may require related objects in the BDU tobe added, deleted, or updated. The related objects that need to beadded, deleted, or updated can be identified and located by followingthe relationships between objects. Once the related objects are found,JEP 300 spawns new jobs to update the related objects.

New relationships between classes can be defined in a user interface asshown in FIG. 7. The user interface displays a schema window 60, whichallows the system administrator to add and delete relationships betweenclasses of objects, for example, an organization class 61, a personclass 62, and a product class 63.

When a new relationship is defined, each object in one class must belinked to a corresponding object in another class. Similarly, when a newobject is created by a JCP 350, new relationships between the new objectand other existing objects must be established. To locate the existingobjects in a relationship, JCP 350 uses an index for all the objects inBDU 22. From the information stored in the schema, JCP 350 knows whichindex to select and how the information is sorted within the index. TheJCP creates another job for establishing a relationship between eachexisting object and the new object.

To establish a relationship between objects that may be distributedacross multiple processors and databases, additional jobs and objectshave to be created to manage the message-passing between objects andsynchronous operations. More specifically, a relationship may beimplemented as a set of interconnected role objects, one role object foreach class. FIG. 10(1)-(4) and FIG. 6 illustrate the process forestablishing relationships for a newly created object 1 with existingobject 2 and object 3. Object 1, object 2 and object 3 are instances ofclass 1, class 2 and class 3, respectively, and the objects are shown inFIG. 9 as C1, C2 and C3, respectively.

First, a role object R1 is created by a job J1 for object C1 (510 and620). Then jobs J1 a ^(t) and J1 b ^(t) are created and sent to C2 andC3 (622), each with a pointer pointing to R1 (520). The superscript ‘t’indicates that J1 a ^(t) and J1 b ^(t) carry a tag and a quorum fractionfor spawning synchronous jobs. J1 a ^(t) and J1 b ^(t) create roles R2and R3 (640 and 660), and send pointers (531, 532) connecting R2 and R3back to R1, respectively.

J1 a ^(t) and J1 b ^(t) further spawn synchronous jobs J1 a 1 ^(s) andJ1 b 1 ^(s) (530, 642 and 662), and send them back to R1 (643 and 663).The superscript ‘s’ indicates that J1 a 1 ^(s) and J1 b 1 ^(s) aresynchronous jobs, such that neither J1 a 1 ^(s) nor J1 b 1 ^(s) mayexecute until both are ready to run. Before running, J1 a 1 ^(s) and J1b 1 ^(s) are collapsed into a single job, which contains informationabout R2 and R3 carried by J1 a 1 ^(s) and J1 b 1 ^(s), respectively.The information includes the pointers that point to R2 and R3 (531 and532), and pre-determined cache information of C2 and C3, which will bedescribed later. The single job records the pointers and caches thepre-determined cache information in R1 (624).

After the single job completes, it spawns final creation jobs J2 a andJ2 b and sends them to R2 and R3 (626), respectively, with theinformation of R1, R2 and R3 (540). R2 and R3 use the information torecord the pointers of the other two (541, 542, 543 and 544) and cachethe information about the other two, respectively (644 and 664). Therelationship is not available to an object until its role has theinformation of all of the other roles (680).

After a relationship is established, a user of the system may wish tosee all the relationships of an object to be displayed, together withcertain information about the other objects participating in therelationships. To increase the performance of displaying theinformation, the role of the object caches information about otherobjects with which its object has relationships. For example, a personmay have many relationships to other people, products and organizations,which are usually scattered across multiple databases. It is inefficientto retrieve information about the scattered objects across multipledatabases. Therefore, role objects cache information from the otherobjects in the relationship.

FIG. 11 illustrates a user interface 80 that allows a user to selectcache variables to be cached in a role object participating in anownership relationship. The user may indicate the cache variables bymarking the attributes in a column 81 labeled as “Data” on the top. Asummary of all relationships of an object, including the cachedinformation about other objects in the relationships, can be quicklydisplayed in a list.

Every role has a version number that increases when its associatedobject is modified. When the version number of the object is changed, amessage is sent to the other roles of the object's relationships so thatthe values of the object cached in the other roles can be updatedaccordingly. The version number cycles back to 0 every 65536 versions.

Every role also tracks the version of all other roles that it currentlyhas cached, and the number of versions missing for each other role. Aversion may be missing because messages containing version numbers maybe delayed for variable lengths of time during transmission over anetwork, thus causing out-of-order reception. The number of missingversion numbers for each other role indicates how many outstandingmessages from that role are yet to be received. A role may not want todelete itself if outstanding messages are about to arrive.

To compute the number of missing versions, the role takes the receivednew version number and subtracts the current version number. Thedifference minus one is added to a running total that indicates thenumber of missing versions. When a version less then the current versionis received, the difference between the current version and the receivedversion is computed, and the running total of missing versions isdecremented by one. For example, if the current version is 6 and aversion 10 arrives, we record the fact that 10−6−1=3 versions are stillexpected (7,8,9). After version 10 has arrived, receiving old version 8means there are still 2 old versions in transit (7 and 9).

A relationship may be deleted as a result of an associated object beingdeleted or updated. It is also possible to delete a relationship becauseit is no longer necessary. When a relationship is deleted betweenobjects, an algorithm for the relationship deletion guarantees thecorrectness of the deletion even in the presence of simultaneous deleterequests from different objects in the relationship. The algorithmguarantees that there will never be a message arriving for a role thathas been physically deleted even though the USP does not guarantee theorder in which the messages arrive.

The deletion process begins when an object tells one of its roles todelete that role's relationship. This role is called the initiator. Atschema definition time, one of the role classes of the relationship isarbitrarily selected as the coordinator role. The coordinator is allowedto be the initiator.

If the initiator is already marked for deletion, it indicates thatdeletion is already in progress and the relationship will eventually bedeleted. Thus, the initiator does nothing. Alternatively, if theinitiator has not been marked for deletion, it marks itself for deletionand sends a message 1 to the coordinator role. The final version numberof the initiator is passed along in the message 1. The version number isused to order role cache update requests (i.e., when an object changes,all roles that participate in relationships with the object's role areasked to update their caches with the new information). Because it ismarked for deletion, the initiator role ignores subsequent changes tothe initiator role's object, and does not send change messages to theother roles.

When the coordinator receives a message 1, it increments a counterindicating how many neighboring roles have been marked as deleted. Ifthis was the first such message, a message 2 is sent to each role.

When message 2 is received by a role, the deletion flag is examined. Ifthe role is already marked for deletion then it means that a message 1was already sent to the coordinator from this role. So the role simplyrecords that the message 2 has arrived and sends no reply. Otherwise therole marks itself as deleted and sends a message 1 to the coordinator toindicate this.

These rules for messages 1 and 2 guarantee that the coordinator willreceive exactly one message 1 from each role, and will receive thatmessage only after that role has been marked deleted. This is true evenif there are multiple initiators, each attempting to trigger deletion ofthe relationship.

When the counter in the coordinator indicates that all roles have beenmarked as deleted (because the coordinator has received a message 1 fromeach role), the coordinator sends a message 3 to each role to indicateit is safe to physically delete it.

These message 3's are the last messages sent to the roles from thecoordinator. Since each role was already marked as deleted prior tothis, they have also stopped sending cache-updating messages to eachother. However, there may be messages that were sent long ago that stillhave not arrived (because the USP does not guarantee ordering ofmessages). To avoid physically deleting a role before all messages havearrived at it, each role has an array of version numbers, one for eachother role. The version number records the latest version number amongthe received messages for the corresponding role. Another arraymaintains an outstanding message count for each other role, the countindicating that how many messages have not yet arrived from each otherrole. The outstanding messages are typically cache-updating messages.

The algorithm guarantees only one message 3 will ever arrive at a role,and it carries an array of final version numbers for all the roles. Whenthis message arrives, a ready-to-physically-delete flag is set. If thecounters inside the role indicate that there are no outstanding incomingmessages, the role is immediately deleted. Otherwise, whenever an oldcache-update message finally arrives at the role, the counters areupdated and, if they indicate all messages have arrived and the role ismarked as ready-to-physically-delete, the role is physically deletedfrom the database.

Message 2 can arrive at a role after message 1, if the role is aninitiator. A flag in each role indicates whether the message 2 hasarrived yet, and physical deletion is postponed until the message 2 hasarrived (as well as any outstanding cache-updating messages, asdescribed above).

The following is a brief summary of the information contained in thethree types of messages:

-   -   Message 1 (“A role has been marked for deletion.”) contains:        -   The role that was marked for deletion.        -   The final version number of that role.    -   Message 2 (“Please mark for deletion on behalf of coordinator.”)        contains:        -   The coordinator role's final version number.    -   Message 3 (“Physically delete role when old messages are all        accounted for.”) contains:        -   The final version number of each role.

At the moment a role is marked as deleted, that role should bedisconnected from its object. Thus, from the viewpoint of the object, itappears that the deletion has already happened.

As an example, consider three connected roles, R1, R2, and R3, where R2is the coordinator. Referring to FIG. 12 and FIG. 13( a)-(f), supposethat the deletion is initiated at R1 (810, 820). Also assume that thereis an outstanding cache-updating message from R1 to R3 that is intransit for the entire example. The example reflects the steps taken byeach of the roles.

-   R1: I'm not yet marked (811), so I'll mark myself deleted (813) and    send a message 1 to R2 (814), the coordinator. It will contain my    final version number, FV1.-   (Suppose that there are no cache-updating messages in transit from    R1 to R2.)-   R2: Receiving message 1 from R1 (830), I record in my table of role    version numbers that FV1 is the current version for R1 (835). I see    that there are no cache-updating messages in transit from R1 to R2.    I now send out a message 2 to each role (R1, R2, and R3) (837). This    message contains my final version number FV2.-   R1: I receive the message 2 (831), but since I already marked myself    as deleted, I simply record the coordinator's (R2's) final version    number.-   R2: I receive the message 2 (831). Since I have not yet marked    myself deleted (832), I mark myself deleted (833) and send a message    1 to the coordinator (i.e., myself) (834), including my final    version number FV2.-   R3: I receive the message 2 (831). Since I have not yet marked    myself deleted (832), I mark myself deleted (833) and send a message    1 to the coordinator (R2) (834), including my final version number    FV3.-   (Suppose that R2 receives message 1 from R3 before it receives    message 2 from R2.)-   R2: I receive message 1 from R3 first. I record R3's final version    number in my array of current versions (835). Since I have only    received two message 1's (from R1 and R3), I do nothing else.-   R2: I receive message 1 from R2 next (831). Since this was my 3rd    message 1, I now know all final version numbers of all roles, as    well as the fact that they're all marked for deletion. Therefore I    send a message 3 to each role (838), passing the final version    numbers FV1, FV2 and FV3 in each message.-   (Suppose that after R1, R2 and R3 receive message 3 from R2, there    is no outstanding message for R1 and R2, but one outstanding message    for R3.)-   R1: I receive message 3 from R2, indicating I can physically delete    myself I reconcile the final version numbers against my current    versions (839). That is, I check for outstanding messages in my    array of outstanding message counts, I see that there are none.    Therefore I delete myself (840).-   R2: I receive message 3 from R2, indicating I can physically delete    myself I reconcile the final version numbers against my current    versions (839). That is, I check for outstanding messages in my    array of outstanding message counts, I see that there are none.    Therefore I delete myself (840).-   R3: I receive message 3 from R2, indicating I can physically delete    myself I reconcile the final version numbers against my current    versions (839). That is, I check for outstanding messages in my    array of outstanding message counts, I see that there is one    outstanding cache-updating message from R1. I mark myself as    ready-to-physically-delete and wait for the next message (841).-   R3: I receive the final outstanding cache-updating message from R1    (842), note that it arrived, and notice that it was the last message    I was waiting for and that my ready-to-physically-delete flag is set    (839). I then physically delete myself from the database (840).

Referring again to FIG. 9, messages for deleting a relationship maysometimes arrive when a role is in the process of creating therelationship. To prevent a message from being sent to a non-existentrole, the role will complete the creation job before it deletes itself.If a role receives a deleted message before it has received the finalcreation job (J2 a or J2 b), it will mark itself as deleted and waituntil the final creation job is received. As soon as the final creationjob is received, the role will proceed with processing the deletemessage.

Appendix A contains source code of an implementation of the inventionfor use on a system in which VisualWorks SmallTalk 5i.1 is installedwith an Objectivity/DB 5.2.2 database system.

Other embodiments are within the scope of the following claims. Forexample, the invention could be implemented on a database that is not anobject database, such as a relational database. In an object database,the data objects can be referred to as data items, and the data objectattributes can be referred to as data elements. In a relational databasethe data records could be considered the data items and the data fieldscould be considered the data elements.

1-23. (canceled)
 24. A method comprising storing data items of adatabase persistently in a first physical cluster of a storage medium,storing in other physical clusters of a storage medium data items thatare replicas of the data items of the database that are stored in thefirst physical cluster, partitioning the stored data items and thestored replicas of the data items into partitions that enable analgorithm that uses the data items to be decomposed into sub-algorithmsthat can be processed concurrently, and maintaining consistency betweenthe data items and the replicas of the data items as the data items orthe replicas of the data items are updated.
 25. The method of claim 24in which each replica contains only data that is required by one of thesub-algorithms.
 26. The method of claim 24 in which the selection ofdata for each of the replicas is a selection of items of data, or ofelements of items, or of both.
 27. The method of claim 24 in which thereplicas are provided persistently as objects to an object-orientedapplication.
 28. The method of claim 27 in which an object relationalbroker provides persistent storage of objects for an object-orientedapplication.
 29. The method of claim 24 in which the data is stored inan object database.
 30. The method of claim 24 in which the data isstored in a relational database with object-oriented extensions.
 31. Themethod of claim 24 in which consistency between the data items and thereplicas is maintained by concurrent processing on at least twodifferent processors.
 32. The method of claim 31 in which the twodifferent processors do not use shared memory for the concurrentprocessing.
 33. The method of claim 24 in which, while one of thereplicas is being updated, the related portions of the data may beconcurrently updated.
 34. The method of claim 24 in which the data itemsin the replicas can be concurrently updated by at least two differentprocessors.
 35. The method of claim 24 in which the portions of the dataincluded in each of the replicas are specified by a user.
 36. The methodof claim 24 in which the portions of the data included in each of thereplicas are selected.
 37. The method of claim 36 in which the selectionapplies to items of the data.
 38. The method of claim 36 in which theselection applies to elements of items of the data.
 39. The method ofclaim 36 in which the selection applies to items of data and to elementsof items of the data.
 40. The method of claim 36 in which the selectionis based on different criteria for different replicated sets.
 41. Themethod of claim 24 in which the replicas are physically clustered instorage.
 42. The method of claim 24 in which the replicas arerespectively physically clustered.
 43. The method of claim 24 in whichthe replicas are associated with indexes.
 44. The method of claim 43 inwhich the indexes include replicated data for use by the algorithm. 45.The method of claim 44 in which the indexes are physically clustered instorage.
 46. The method of claim 24 in which the concurrent processingis done in a predetermined relative order with respect to the storeddata and the replicas.
 47. The method of claim 24 in which the storeddata includes data items of the database that comprise objects in anobject database.
 48. The method of claim 24 in which the stored dataincludes data items that are provided as objects to an object-orientedapplication.
 49. The method of claim 24 in which an object relationalbroker provides persistent storage of objects for an object-orientedapplication.
 50. The method of claim 24 in which maintaining theconsistency includes maintaining and making available information abouttemporarily incomplete states.
 51. The method of claim 50 in which theincompleteness of the states is communicated to processes that seek toupdate at least one of the items or replicas.
 52. The method of claim 24in which the data items or replicas are accessed from an object-orientedapplication.
 53. The method of claim 24 in which two data items orreplicas are stored in two different clusters and the consistency isguaranteed by maintaining relationship roles associated respectivelywith the two data items or replicas.
 54. The method of claim 24 in whichthere are more than two data items or replicas and the relationshipsamong them are binary, ternary, or N-ary.
 55. The method of claim 24 inwhich there are more than two data items or replicas and therelationships among them are one-to-one or one-to-many or many-to-many.56. The method of claim 55 in which the relationships among the dataitems and the replicas are bi-directional.
 57. The method of claim 24 inwhich relationships among the data items and the replicas are maintainedby representing the relationships by roles, each of the roles beingassociated with one of the data items or replicas.
 58. The method ofclaim 24 in which relationships among the data items and the replicasare established by concurrent processing with respect to the data itemsin the replicas.
 59. The method of claim 24 in which relationships amongthe data items in the replicas are created by creating a set of roleobjects, one for each of the data items for which a relationship is tobe created, the role objects including pointers to one another.
 60. Themethod of claim 59 in which the creations of the role objects aresynchronized.
 61. The method of claim 59 in which the role objectsinclude replicated information about data items or replicas with whichthe data item or replica associated with the role object has therelationship.
 62. The method of claim 61 in which the role objectincludes a version number that is incremented each time the associateddata item or replica is updated.
 63. The method of claim 61 alsoincluding sending messages among the role objects to keep replicateddata and version number information current among the role objects. 64.The method of claim 61 in which role objects of relationships storeversion numbers of the other role objects with which they haverelationships.
 65. The method of claim 61 in which the role objectswhich have relationships store information about missing versions forthe other role objects with which they have relationships.
 66. Themethod of claim 61 also including assuring that, when a relationship isdeleted, the deletion is correct notwithstanding multiple pending deleterequests from different objects in the relationship.